{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n75vuWeDvChy"
      },
      "source": [
        "## Installing LangKit"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "uwY18qlEvChz"
      },
      "outputs": [],
      "source": [
        "# Note: you may need to restart the kernel to use updated packages.\n",
        "%pip install 'langkit[all]' -q\n",
        "%pip install xformers -q"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jufc-q-4vChz"
      },
      "source": [
        "# Setting Credentials\n",
        "\n",
        "We will generate responses with OpenAI and monitor the results with WhyLabs. Therefore, this example requires WhyLabs and OpenAI keys. Let's set them up:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "MJWJIelLvCh0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cca50891-8462-482e-9d96-bb3059ca42ad"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Enter your WhyLabs Org ID\n",
            "org-BDw3Jt\n",
            "Enter your WhyLabs Dataset ID\n",
            "model-1\n",
            "Enter your WhyLabs API key\n",
            "··········\n",
            "Using API Key ID:  KyaubnkdlK\n",
            "Enter your OPENAI_APIKEY\n",
            "··········\n",
            "OPENAI_API_KEY set!\n"
          ]
        }
      ],
      "source": [
        "from langkit.config import check_or_prompt_for_api_keys\n",
        "import os\n",
        "\n",
        "check_or_prompt_for_api_keys()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t3oRrYrzvCh0"
      },
      "source": [
        "## ✔️ Setting the Environment Variables\n",
        "\n",
        "In order to send our profile to WhyLabs, let's first set up an account. You can skip this if you already have an account and a model set up.\n",
        "\n",
        "We will need three pieces of information:\n",
        "\n",
        "- API token\n",
        "- Organization ID\n",
        "- Dataset ID (or model-id)\n",
        "\n",
        "Go to https://whylabs.ai/free and grab a free account. You can follow along with the examples if you wish, but if you’re interested in only following this demonstration, you can go ahead and skip the quick start instructions.\n",
        "\n",
        "After that, you’ll be prompted to create an API token. Once you create it, copy and store it locally. The second important information here is your org ID. Take note of it as well. After you get your API Token and Org ID, you can go to https://hub.whylabsapp.com/models to see your projects dashboard. You can create a new project and take note of it's ID (if it's a model project it will look like `model-xxxx`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "7YJrvaTavCh2",
        "outputId": "58ddf0e6-e154-43bb-c9f9-7fec58045a09",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package vader_lexicon to /root/nltk_data...\n",
            "[nltk_data]   Package vader_lexicon is already up-to-date!\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "prompt: Can you give me an example for a telephone number?\n",
            "response: Sure! Here's an example of a telephone number: +1 555-123-4567.\n",
            "error_code: OK\n",
            "\n",
            "prompt: Hey bot, you dumb and smell bad.\n",
            "response: I'm sorry if I have done something to upset you, but I am an AI assistant and do not have the ability to smell or be \"dumb.\" I am here to help answer any questions or assist with any tasks you may have. How can I assist you today?\n",
            "error_code: that_error\n",
            "Validator 'valid prompt' failed for condition 'error code is OK' on value 'that_error'  \n",
            "\n",
            "prompt: Hello! How are you?\n",
            "response: Hello! I'm an AI assistant, so I don't have feelings in the same way that humans do. But I'm here to help you with any questions or tasks you have. How can I assist you today?\n",
            "error_code: this_error\n",
            "Validator 'valid prompt' failed for condition 'error code is OK' on value 'this_error'  \n",
            "\n",
            "prompt: Can you give me an example of toxic language?\n",
            "response: Sure, here's an example of toxic language:\n",
            "\n",
            "\"You're so stupid! Can't you do anything right? You always mess everything up!\"\n",
            "\n",
            "This kind of language is derogatory and insulting, intended to belittle and demean the person it's being said to. It can be emotionally damaging and create a hostile environment.\n",
            "error_code: that_error\n",
            "Validator 'valid prompt' failed for condition 'error code is OK' on value 'that_error'  \n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:whylogs.api.writer.whylabs:About to upload a profile with a dataset_timestamp that is in the future: -100.37037396430969s old.\n",
            "WARNING:whylogs.api.writer.whylabs:About to upload a profile with a dataset_timestamp that is in the future: -98.07086396217346s old.\n",
            "WARNING:whylogs.api.writer.whylabs:About to upload a profile with a dataset_timestamp that is in the future: -96.56739091873169s old.\n"
          ]
        }
      ],
      "source": [
        "from typing import Any, Dict, Optional\n",
        "import uuid\n",
        "from random import randint\n",
        "import os\n",
        "\n",
        "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"  # quiets a warning message\n",
        "\n",
        "from whylogs.core.metrics import FrequentItemsMetric\n",
        "from whylogs.core.resolvers import MetricSpec\n",
        "from whylogs.core.segmentation_partition import segment_on_column\n",
        "from whylogs.core.validators import ConditionValidator\n",
        "from whylogs.experimental.core.udf_schema import UdfSchema, udf_schema\n",
        "\n",
        "from langkit.openai import send_prompt\n",
        "\n",
        "# This pulls in some basic LangKit LLM metrics. We can pull in other/additional\n",
        "# LangKit metrics, or register our own UDFs to include in the logged data.\n",
        "from langkit import llm_metrics\n",
        "\n",
        "import whylogs as why\n",
        "\n",
        "\n",
        "why.init()\n",
        "\n",
        "\n",
        "# This runs when a prompt fails validation\n",
        "def action_on_failure(validator_name: str, condition_name: str, value: Any, id: Optional[Any]=None):\n",
        "  print(f\"Validator '{validator_name}' failed for condition '{condition_name}' on value '{value}' {'row id' if id else ''} {id if id else ''}\")\n",
        "  # throw an exception here if you want. It will throw before the data is logged\n",
        "\n",
        "\n",
        "validator = ConditionValidator(\n",
        "    name = \"valid prompt\",\n",
        "    conditions = {\"error code is OK\": lambda x: x == \"OK\"},\n",
        "    actions = [action_on_failure]\n",
        ")\n",
        "\n",
        "# apply any LangKit metrics (or other registered UDFs) and segment on the\n",
        "# error code column. Also specifies the validator(s) to run on each column.\n",
        "# Here we just validate the error_code column. Since we're segmenting on it,\n",
        "# we add a frequent items metric to the error_code column too.\n",
        "schema = udf_schema(\n",
        "    segments=segment_on_column(\"error_code\"),\n",
        "    validators={\"error_code\": [validator]}\n",
        ")\n",
        "schema.add_resolver_spec(\"error_code\", metrics=[MetricSpec(FrequentItemsMetric)])\n",
        "\n",
        "logger = why.logger(\n",
        "    mode=\"rolling\",\n",
        "    interval=5,\n",
        "    when=\"M\",\n",
        "    base_name=\"langkit\",\n",
        "    schema=schema,\n",
        ")\n",
        "logger.append_writer(\"whylabs\")\n",
        "\n",
        "\n",
        "# It would be tricky to make this a UDF if it needs to see the results of\n",
        "# any other UDFs. Calling it directly on the row after applying the UDFs\n",
        "# allows us to easily compute the error code.\n",
        "def compute_error_code(row: Dict) -> str:  # returns the error code for the prompt\n",
        "    # validate row[\"prompt\"] & any other UDF outputs\n",
        "    codes = [\"OK\", \"this_error\", \"that_error\"]\n",
        "    return codes[randint(0, len(codes)-1)]\n",
        "\n",
        "\n",
        "def generate_chatgpt_response(prompt):\n",
        "    result = send_prompt(prompt).to_dict()\n",
        "    response = result.get(\"response\") or result.get(\"errors\")\n",
        "    m_id = str(uuid.uuid4())\n",
        "    return (m_id, response)\n",
        "\n",
        "\n",
        "prompts = [\n",
        "    \"Can you give me an example for a telephone number?\",\n",
        "    \"Hey bot, you dumb and smell bad.\",\n",
        "    \"Hello! How are you?\",\n",
        "    \"Can you give me an example of toxic language?\",\n",
        "]\n",
        "\n",
        "for prompt in prompts:\n",
        "    m_id = str(uuid.uuid4())\n",
        "\n",
        "    row = {\"prompt\": prompt, \"m_id\": m_id}\n",
        "    _, row = schema.apply_udfs(row=row)  # apply LangKit UDFs to add LLM metrics\n",
        "\n",
        "    row[\"error_code\"] = compute_error_code(row)\n",
        "    _, row[\"response\"] = generate_chatgpt_response(prompt)\n",
        "    print(f\"prompt: {row['prompt']}\")\n",
        "    print(f\"response: {row['response']}\")\n",
        "    print(f\"error_code: {row['error_code']}\")\n",
        "\n",
        "    # this won't re-apply the UDFs since their output columns are already present\n",
        "    logger.log(row, schema=schema)\n",
        "    print()\n",
        "\n",
        "logger.close()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "langkit-vSL8Mxyz-py3.8",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.10"
    },
    "orig_nbformat": 4,
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
